ci: hourly full run on main, push narrows to the affected set, and a filer for a red one - #16991

Part of #16467

push on main re-ran the full Test Core battery on a tree the merge queue had just verified — ~90 machine-hours a day, and 36 of the last 60 runs were cancelled by the next merge before finishing. This narrows push to the affected set and puts the full battery on an hourly clock, on both required-context workflows, with a filer that reports a red one.

The three parts land together on purpose. Narrowing push while nothing runs the full battery and nothing files on red would leave main less tested than today and unwatched — strictly worse than not doing the card.

What changed

The two hazards the card did not name

HAZARD 1 — the hourly run would have been cancelled by the next merge. On a schedule event github.event.pull_request.number is empty and github.ref is refs/heads/main, so the old two-segment key produced a group byte-identical to a push to main, under cancel-in-progress: true. Fixed by putting github.event_name in the key, and asserted rather than read: scripts/ci/scheduled-full-run.mjs --check-concurrency evaluates each file's concurrency.group expression against a push-shaped and a schedule-shaped context and reds when they are equal.

scheduled-full-run: .github/workflows/ci.yml OK -- push 'ci-.github/workflows/ci.yml-push-refs/heads/main'
  and schedule 'ci-.github/workflows/ci.yml-schedule-refs/heads/main' are different concurrency groups.
scheduled-full-run: .github/workflows/lint.yml OK -- push 'lint-...-push-refs/heads/main'
  and schedule 'lint-...-schedule-refs/heads/main' are different concurrency groups.

Ablation (mutation confirmed on disk, restored and hash-verified byte-identical): reverting ci.yml's key to the pre-card two-segment form makes both the gate and its self-test red, naming the pair —

scheduled-full-run: .github/workflows/ci.yml: a `schedule` run and a `push` run share the concurrency
  group 'ci-.github/workflows/ci.yml-refs/heads/main', so the next merge to main cancels the hourly
  full run. Put `github.event_name` in the key.
--check-concurrency EXIT=1   --self-test EXIT=1   (both EXIT=0 restored)

The gate also refuses the two degenerate ways to make the groups differ — a run-unique key (github.run_id) and a constant key — because both switch cancel-in-progress off or on for every event in the file. The expression evaluator throws on any token outside its grammar rather than guessing a value.

HAZARD 2 — the timings dataset would have been regenerated from a partial measurement. select-shard-timings-run.mjs hard-coded ?event=push&branch=main, and its eligibility test is "six shard jobs succeeded + artifacts present". Under an affected-only push that test still reads green over a measurement of whatever the last merge touched. coverageReport is not a backstop either: a package the affected set skipped is a turbo cache HIT, carried at its old weight, which is a pass. The selector now names DEFAULT_RUN_EVENT = 'schedule'.

A new self-test battery drives listCandidates against an injected fetchImpl that serves both a schedule-shaped and a push-shaped run, both fully eligible:

• reading — the default call returns only the schedule run, and the request never contains event=push.
• firing control — asking for {event: 'push'} explicitly returns the push run, and that run is judged eligible: true. So the zero above is a reading about the query, not a mute fixture, and eligibility demonstrably cannot tell the two apart.
• nonsense control — an event nobody publishes returns nothing, so the run list is genuinely keyed on the event.

Ablation, behavioural. Setting DEFAULT_RUN_EVENT back to 'push' and neutralising both identity checks, so only the behavioural cases can fire:

Error: default selection: expected only the schedule run, got [100]     EXIT=1

100 is the push-shaped run's id. Restored and hash-verified byte-identical; the self-test is green again.

There is also a ⛔ in shard-timings-refresh.yml that is now load-bearing: ci.yml's Save Turbo cache (main only) step must stay github.event_name == 'push'. If the hourly run seeded the cache it restores, the next hourly run would replay almost the whole workspace, the generator would refuse every replayed task, and this lane would measure nothing — by construction, every hour. Affected-only pushes seeding a narrower cache is the direction that helps: it leaves more real misses for the hourly run to time.

Negative control — pull_request and merge_group selection is byte-identical

Measured, not asserted. The new self-test harness was run twice over one fixture, changing only the script under test: the pre-card select-shard-packages.sh (extracted from the merge base) and the one in this PR. Transcripts were compared per case, with 40-hex shas normalised.

NEGATIVE CONTROL  pull_request + merge_group cases: 19 | identical: 19 | differing: []
FIRING CONTROL    push/schedule/dispatch cases:     12 | identical:  4 | differing: 8

The firing control is what makes the 19 a reading: the same comparison separates 8 of the 12 push/schedule cases. The 4 that match are schedule, workflow_dispatch and an unlisted event, which correctly fell to the full list under both scripts.

The permanent half of that control is in the selftest itself: one pull_request case and one merge_group case are driven with OS_SHARD_PUSH_BEFORE_SHA set to a real, resolvable commit — the value that would move the answer if it leaked — and pinned to the same diff base, the same union input and the same package set.

select-shard-packages.sh on push

Every fallback is the FULL list plus a ::warning::; no branch selects a narrower set quietly.

Battery floor raised 20 cases / 80 checks → 32 / 158, at the measured values.

The filer — a new workflow, and why not an extension

merge-queue-triage.yml was the candidate. It is 911 lines about the merge queue; its entire if: is event == 'merge_group', its subject is a PR comment, and its workflows: [CI] list would have had to grow a second entry whose events it must then exclude everywhere. Two products with disjoint triggers in one file buys nothing and makes both harder to read. The new file follows test-nightly-tiers.yml's shape instead.

Two cards, not one. CI and Lint & Type Check go red for unrelated reasons and the body is rewritten on a refresh, so a shared identity would make whichever filer ran second overwrite the other's diagnosis. Identity is the workflow name; "one red scheduled run files exactly one card" holds per run, which is the unit that is red.

De-dup is tested, not described. Identity, matching, paging and the whole create-or-refresh decision live in scheduled-full-run.mjs and are driven offline — a workflow_run workflow only ever runs from the default branch, so nothing on a PR can exercise the live path. The acceptance property is driven end to end against a mutable board:

• run 1, empty board → filed, board holds 1 card, labels applied on create.
• run 2 → refreshed, same number, board still holds 1 card, body rewritten.
• the other workflow going red in the same hour → filed (its own card), board holds 2, and CI's body untouched.
• CI's third red, with both cards open → refreshed, still 2 cards.

Plus the things that are not this card: a pull request carrying the title, the sibling workflow's card, an unrelated issue. And a scan that hits its page bound throws instead of filing — a truncated scan has not established absence, and filing on an unestablished absence is how a filer mints a duplicate an hour.

The body carries the run link and the commits between the previous green schedule run of that workflow and this one. An empty range is its own reading — same tree, green then red, so it is a flake or infrastructure and nobody should go hunting a commit — and a failed lookup is deliberately not allowed to fall through to that sentence.

Two things the card did not mention that had to change

1. dorny/paths-filter's if: had to become an allow-list. It was github.event_name != 'merge_group'. The action resolves a diff only on pull_request and push; on schedule it would have run with no base, every filter output would have come back false, and || 'true' supplies nothing when the output exists and says false — so every downstream != 'false' guard would have skipped. The hourly full run would have been an entirely green, entirely empty run of nothing at all. It is now event_name == 'pull_request' || event_name == 'push', so each new event fails toward THE FILTER CONTRACT's half 1 instead.
2. --limit 15 → 24 in the refresh workflow. The window that matters is the 1-day artifact retention, which now holds ~24 hourly runs; coverage is accumulated across runs, so examining fewer than the window holds is coverage left on the table.

lint.yml needs no step changes: select-gate-families.sh already runs every family for any event that is neither merge_group nor pull_request.

Verification

Gate families derived with node scripts/pm/dispatch-gates.mjs --commands (no hand-written path list) and reconciled with --ran, at final head f6a2737a62:

Run reconciliation - 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN.
dispatch-gates --ran: 63 derived famil(ies) accounted for.

58 exit 0. 5 exit 3 = PREREQUISITE NOT MET, not red, all from an unbuilt workspace: check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content, check:type-check-debt (--re-measure needs 32 built dependency entry points). This diff contains zero packages/** source, so none of them can move; CI builds and runs them. Reported as NOT MEASURED rather than as a pass.

Repo-scoped lint ran in full — no narrowing was needed:

pnpm exec eslint . --no-inline-config --format json    EXIT=0
files: 6390   errors: 0   warnings: 0

6390 is the receiving population as eslint resolved it from eslint.config.mjs, read out of its own JSON reporter, at final head f6a2737a62.

Directly affected families, re-run after the last content change, all exit 0: scheduled-full-run --self-test / --check-concurrency, select-shard-timings-run --self-test, check:select-shard-packages, check:select-gate-families, check-self-test-wired, check-self-test-workflow-commands, check-ci-filter-parity, check:required-contexts, check:nul-bytes, check:workflow-status-functions, check:pm-dispatch-gates, check-step-collectors, check-scripts-symbol-anchors, check-whole-set-label-write, check:watch-hint-literal, check:declared-population-live, check-comment-mask-corpus.

Control-character sweep over every changed file (grep -naP for the C0 set plus DEL): no matches.

Patch round — zero candidate runs is NOT MEASURED, not a finding

Regenerate the shard-timings dataset went red on the first push of this branch, and the number in its log is the whole story:

select-shard-timings-run: NO ELIGIBLE RUN among the 0 most recent completed `schedule` runs of ci.yml on main.
  Every one was censored, failed, or has lost its run-summary artifacts to the 1-day retention window.

0. Nothing was examined — schedule: does not exist on main until this merges — and the sentence named three causes, none of which had occurred. It also would not have cleared at merge: there is a bootstrap window of at least an hour before the first hourly run finishes and is downloadable.

⛔ Not fixed by falling back to event=push; that is the partial-measurement hazard this card exists to close. Fixed with the idiom #16558 / PR #16970 landed: EXIT_PREREQUISITE_NOT_MET (3) from scripts/import-prerequisite.mjs, imported rather than re-picked.

shard-timings-refresh.yml reads 3 as NOT MEASURED: it does not regenerate, does not commit, does not fail — generate and compare carry if: steps.select.outputs.not_measured != 'true', and everything after compare is already gated on steps.compare.outputs.changed, which is the empty string when compare is skipped.

⛔ And it is never quiet. A ::warning:: annotation, a step-summary section and the selector's own refusal text, all carrying: a PERSISTENT NOT MEASURED is a defect, not a steady state — if it is still saying zero once the hourly run has been live a few hours, the trigger is gone or every run is being cancelled, and the dataset is ageing out while the job reports green.

The consumer is driven, not read

The step's run: block is lifted out of the YAML by extractStepScript and executed under bash -e against a stub node that answers with the exit code each case is about — three legs, because a branch tested only on the path it was written for is a branch nobody has seen fail:

• 3 → step exits 0, not_measured=true, an annotation is minted, and the summary carries the persistent-is-a-defect sentence and the selector's verbatim refusal.
• 1 → step exits 1, ::error::, and not_measured is not set. A finding is still a finding.
• 0 → step exits 0, no skip flag, the rest of the job runs.

Plus a nonsense control: extracting a step name that is not in the file throws rather than returning an empty script that would make every assertion vacuous. And the YAML half is pinned — both generate and compare must carry the guard.

Ablations (each mutation confirmed on disk by counting both spellings; each restored and git hash-object-verified byte-identical):

workflow: `-eq 3` -> `-eq 99`      self-test EXIT=1  "workflow leg 3: the step failed (exit 3)"
script:   empty-leg guard disabled  self-test EXIT=1  "empty leg: expected exit 3, got 1"

Battery floors: new 'select-shard-timings-run empty vs ineligible': 17 and 'shard-timings-refresh NOT MEASURED path': 18, SELF_TEST_BATTERY_FLOOR 2 → 4, all at measured values.

Confirmed on the runner, not only offline

Regenerate the shard-timings dataset on head 428c04dbcb: success, job 102289511470. Its log takes the new path —

select-shard-timings-run: PREREQUISITE NOT MET -- the API returned NO completed `schedule` run
  of ci.yml on main at all (asked for the 24 most recent).
  NOTHING was measured. This is NOT "every candidate was rejected": there were no candidates,
  so no run was censored, none failed, and none lost its artifacts.
  ⛔ PERSISTENTLY NOT MEASURED IS A DEFECT, NOT A STEADY STATE. ...
##[warning]Shard timings NOT MEASURED: no completed `schedule` run of ci.yml exists on main yet. ...

Every check on that head has settled: 31 success, 2 skipped, 0 failing (the two skips are Check Changeset, skipped by skip-changeset, and the opt-in packed-tarball smoke). dispatch-gates --ran at that head: 63 derived, 63 run, 0 UNRUN — 58 exit 0, the same 5 exit-3 PREREQUISITE NOT MET from the unbuilt workspace. pnpm exec eslint . --no-inline-config EXIT=0 over 6390 files, 0 errors, 0 warnings.

Landing

Authored in Claude Code session session_012GKcPZbMoGq7WPzKLfRBTU.

Draft. .github/workflows/** and scripts/ci/** only — no governed surface, no published package moves, so skip-changeset applies. Landing is the PM's.

Generated by Claude Code